Methods and systems for performing navigation-assisted medical procedures

ABSTRACT

Systems and methods are described for performing navigation-assisted medical procedures such as biopsies, surgeries and pathology procedures by obtaining location information of an item of interest located within at least a portion of a subject; sensing position information of a moveable device; determining a relative position of the moveable device to the item of interest using the location information of the item of interest and the position information of the moveable device; and providing feedback based on the relative position of the moveable device to the item of interest that can be used to change the relative position of the moveable device to the item of interest.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional application of U.S. patent applicationSer. No. 14/549,258 filed Nov. 20, 2014 (PG-Publication No.: US2016/0143693 A1, published May 26, 2016), which is fully incorporated byreference and made a part hereof.

BACKGROUND

When performing medical procedures such as surgery, obtainingpathological samples, biopsies, and the like, medical personnel stilluse methods such as palpation, sight and static two-dimensional imagingto locate and sample or review items of interest when performing suchprocedures. Such reliance upon the skill and experience of the medicalpersonnel and dimensionally-limited imaging of the item of interest mayresult in reduced quality and accuracy of the medical procedure. Forexample, pathologic staging of solid tumors involves determining thepresence and extent of disease. Precise specimen processing is desiredfor establishing patient customized management plans that may indicatethe need for post-operative chemotherapy and/or radiation therapy.Failure to identify malignant involvement in tissues can lead to themisdiagnosis and mismanagement of patients, leading to undesirableoutcomes. Furthermore, correctly determining the extent of diseaseinvolves accurate assessment of the tumor's size, and the presence orabsence of metastatic disease involving lymph nodes and distant tissues.Hence, the pathologist's assessment is crucial, and it plays a key rolein basing future treatment options for patients.

Similarly, surgery has offered the best opportunity of a cure forpatients with solid malignancies. Furthermore, the best surgery is thefirst surgery. Optimal surgical approaches to cancer resection in thepast were limited to visual and tactile cues in identifying the tumor'slocation and extent. Surgical procedures were based upon surgicalanatomy and traditional planes of resection that cancer cells disregard.Hence, it is desirable that the surgical team “see” all of the tumor(s),including metastases, to achieve the best resection and offer thepatient the best possible outcome.

Likewise, biopsies such as fine needle biopsies are performedpredominately using the medical personnel's sense of sight and feeling.Biopsies may be inaccurate or incorrect when the needle or other biopsyinstrument extends through the item of interest and unaffected tissueobtained.

Other forms of medical procedures and surgeries such as, for example,reconstructive surgeries that involve bone replacement also havepredominately relied upon the sight and feel of the surgeon, which couldresult in painful mis-alignment and the need for additional surgeries.

It can be said that a surgeon is as good as his or her tools.Unfortunately, though, because of the reliance upon the sight and feelof the medical professional, it has been found that in cancer surgery upto 50 percent of all resections that residual tumor has been leftbehind. It can take some amount of time for the residual tumor to becomemedically symptomatic. In pathology, even with slice depths of only twoto five millimeters (mm), staging can be incorrect up to 30 percent ofall cases. And, with reconstructive surgery, bone implants can bemisaligned in over 50 percent of all surgeries. Therefore, what areneeded are systems and methods that overcome challenges in the art, someof which are described above.

SUMMARY

In one aspect, a method of performing navigation-assisted medicalprocedures is described. One embodiment of the method comprisesobtaining location information of an item of interest located within atleast a portion of a subject; sensing position information of a moveabledevice; determining a relative position of the moveable device to theitem of interest using the location information of the item of interestand the position information of the moveable device; and providingfeedback based on the relative position of the moveable device to theitem of interest that can be used to change the relative position of themoveable device to the item of interest. Location information for theitem of interest can be obtained pre-operative and intra-operative.

Alternatively or optionally, obtaining location information of the itemof interest can comprise obtaining an image of at least a portion of theitem of interest.

Alternatively or optionally, obtaining an image of at least a portion ofthe item of interest can comprise obtaining a three-dimensional image ofat least a portion of the item of interest, which can comprise obtainingthe three-dimensional image of at least a portion of the item ofinterest using a targeted imaging technique such as one or more ofPET/CT, SPECT/CT, PET MRI, ultrasound, florescence, and the like.

Alternatively or optionally, the targeted imaging technique can compriseinjecting a material into the item of interest or the subject andobtaining a three-dimensional image of at least a portion of the item ofinterest using one or more cameras that detect the injected material.The injected materials comprise one or more of Technetium-99, F18,zirconium 89, iodine 123, iodine 124, iodine 125, iodine 131, copper 64,gallium 67, gallium 68, lutetium 177, xenon 133, indium 111, and thelike.

Alternatively or optionally, obtaining a three-dimensional image of atleast a portion of the item of interest can comprise obtaining thethree-dimensional image of at least a portion of the item of interestusing a non-targeted imaging technique such as one or more of X-Ray,ultrasound, CT, ultrasound, MRI, and the like.

Alternatively or optionally, obtaining location information of an itemof interest located within at least a portion of a subject can compriseobtaining location information of a tumor within at least a portion of asubject, cancerous tissue within at least a portion of a subject, anorgan within a body of the subject, and the like.

Alternatively or optionally, obtaining location information of an itemof interest located within at least a portion of a subject can compriseobtaining location information of the item of interest within a body ofthe subject, location information of the item of interest within anorgan of the subject, or location information of the item of interestwithin a tissue specimen. The organ can contained within a body of thesubject, or it can be removed from a body of the subject.

Alternatively or optionally, one embodiment of the method can furthercomprise inserting the moveable device into the item of interest,wherein the feedback indicates a depth that the moveable device has beeninserted into the item of interest.

Alternatively or optionally, providing feedback based on the relativeposition of the moveable device to the item of interest that can be usedto change the relative position of the moveable device to the item ofinterest further comprises acquiring a real-time image of the at least aportion of the subject, wherein the acquired real-time image isreferenced to the same coordinate system as the location information ofthe item of interest and the position information of the moveabledevice; and displaying the real-time image of the at least a portion ofthe subject further comprising the location information of the item ofinterest and the position information of the moveable devicesuper-imposed on the real-time image of the at least a portion of thesubject. The feedback can also be audible or haptic feedback.

Alternatively or optionally, acquiring a real-time image of the at leasta portion of the subject can comprise acquiring the real-time imageusing at least a pair of stereo electro-optical cameras.

Alternatively or optionally, referencing the acquired real-time image tothe same coordinate system as the location information of the item ofinterest and the position information of the moveable device comprisesplacing one or more fiducials on or around the at least a portion of thesubject. At least one of the one or more fiducials comprise aradioactive fiducial.

Alternatively or optionally, displaying the real-time image of the atleast a portion of the subject further comprising the locationinformation of the item of interest and the position information of themoveable device super-imposed on the real-time image of the at least aportion of the subject can comprise displaying the real-time image ofthe at least a portion of the subject further comprising the locationinformation of the item of interest and the position information of themoveable device super-imposed on the real-time image of the at least aportion of the subject on a display that is visible to a person that ismoving the moveable device.

Alternatively or optionally, the real-time image of the at least aportion of the subject can further comprise the location information ofthe item of interest and the position information of the moveable devicesuper-imposed on the real-time image of the at least a portion of thesubject is displayed to the person using augmented reality glasses thatare worn by the person.

Alternatively or optionally, providing feedback based on the relativeposition of the moveable device to the item of interest that can be usedto change the relative position of the moveable device to the item ofinterest comprises providing one or more of visual, audible or hapticfeedback that is used to move the moveable device closer to the item ofinterest. For example, audible feedback can comprise computer-generatedvoice commands, computer-generated noises that change as the moveabledevice moves closer to or further away from the item of interest, andthe like.

Alternatively or optionally, providing feedback based on the relativeposition of the moveable device to the item of interest that can be usedto change the relative position of the moveable device to the item ofinterest can comprise providing a control signal that is used to providerobotic control of the moveable device.

Alternatively or optionally, sensing position information of themoveable device comprises actively or passively sensing positioninformation of a biopsy needle, a scalpel, a pathology wand, a locatorwand, a bone segment, and the like.

Alternatively or optionally, sensing position information of themoveable device can comprise obtaining position information using athree-dimensional sensor. For example, the three-dimensional sensor canbe located on at least one of the moveable device, eyeglasses of aperson in view of the moveable device, or anywhere the three-dimensionalsensor can sense the moveable device. In one embodiment, the eyeglassescomprise augmented reality eyeglasses.

Alternatively or optionally, determining the relative position of themoveable device to the item of interest using the location informationof the item of interest and the position information of the moveabledevice can be performed using a computing device.

Alternatively or optionally, one embodiment of the method can furthercomprising re-registration of the location information of the item ofinterest located within at least a portion of a subject. For example,re-registration can occur after movement of the item of interest, the atleast a portion of the subject, or the subject or after removal of atleast a portion of the item of interest from the at least a portion ofthe subject.

Alternatively or optionally, re-registration can occur dynamically.

Alternatively or optionally, one embodiment of the method can furthercomprise estimating movement or deformation of the item of interestcaused by movement or palpation of the at least a portion of thesubject, wherein the location information of the item of interest ismodified by the estimate.

Alternatively or additionally, movement of the item of interest withinthe at least a portion of a subject can be modeled using a soft tissuemotion engine and the location information of the item of interested canbe updated using the modeled information. The movement can be caused bypalpation of the at least a portion of a subject or a change in positionof the subject.

In another aspect, a system for performing navigation-assisted medicalprocedures is described. One embodiment of the system comprises a devicethat obtains location information of an item of interest located withinat least a portion of a subject; a sensor that senses positioninformation of a moveable device; and a computing device that receivesthe location information of the item of interest from the device and theposition information of the moveable device and executescomputer-readable instructions that determine a relative position of themoveable device to the item of interest using the location informationof the item of interest and the position information of the moveabledevice and provides feedback based on the relative position of themoveable device to the item of interest that can be used to change therelative position of the moveable device to the item of interest.

For example, the device that obtains location information of an item ofinterest located within at least a portion of a subject can comprise oneor more of a PET/CT scanner, a SPECT/CT camera, a PET MRI system, aflorescence detection system, an X-Ray system, an ultrasound scanner, aCT, an MRI, and the like. The item of interest can be one or more of atumor within at least a portion of a subject, cancerous tissue within atleast a portion of a subject, a tissue sample of the subject, an organwithin a body of the subject, a section of bone for bone replacementsurgery, and the like. In one embodiment, the moveable device isinserted into the item of interest and the computing device providesfeedback that indicates a depth that the moveable device has beeninserted into the item of interest.

One embodiment of the system further comprises one or more cameras incommunication with the computing device. The cameras acquire a real-timeimage of the at least a portion of the subject, wherein the acquiredreal-time image is referenced to the same coordinate system as thelocation information of the item of interest and the positioninformation of the moveable device. A display in communication with thecomputing device displays the real-time image of the at least a portionof the subject and the location information of the item of interest andthe position information of the moveable device is super-imposed on thereal-time image of the at least a portion of the subject on the display.The display can comprise augmented reality glasses.

It should be understood that the above-described subject matter may alsobe implemented as a computer-controlled apparatus, a computing system,or an article of manufacture, such as a computer-readable storagemedium.

Other systems, methods, features and/or advantages will be or may becomeapparent to one with skill in the art upon examination of the followingdrawings and detailed description. It is intended that all suchadditional systems, methods, features and/or advantages be includedwithin this description and be protected by the accompanying claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The components in the drawings are not necessarily to scale relative toeach other. Like reference numerals designate corresponding partsthroughout the several views.

FIG. 1 is a flow diagram illustrating example operations for performingnavigation-assisted medical procedures.

FIG. 2 is another flow diagram illustrating example operations forperforming navigation-assisted medical procedures.

FIG. 3 illustrates yet another embodiment of a flow diagram illustratingexample operations for performing navigation-assisted medicalprocedures.

FIGS. 4A and 4B illustrate deformation of the portion of a subject andmovement or deformation of the item of interest located within theportion of a subject.

FIG. 5 illustrates an exemplary process diagram according to embodimentsof the present invention.

FIG. 6 is a block diagram of an example computing device upon whichembodiments of the invention may be implemented.

FIG. 7 illustrates an exemplary overview system for performingnavigation-assisted medical procedures.

FIG. 8 illustrates another embodiment of a system for performingnavigation-assisted medical procedures further comprising one or morecameras in communication with the computing device.

FIGS. 9A-9E illustrate an exemplary integrated process for targetedspecimen analysis.

FIGS. 10A-10C illustrate a phantom bench top study used to show theaccuracy of embodiments of the invention.

FIG. 11 illustrates one type of setup that can be used to performaspects of embodiments of the invention.

FIGS. 12A-12H illustrate an integrated methodology and schema forimage-guided cancer surgery.

FIG. 13 illustrates an embodiment of a system for operating room (OR)setup for intra-operative imaging and 3D recovery and tracking.

FIG. 14 is an exemplary flow diagram that illustrates the integratednature and use of embodiments of the present invention in a medicalapplication.

DETAILED DESCRIPTION

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art. Methods and materials similar or equivalent to those describedherein can be used in the practice or testing of the present disclosure.

As used in the specification and the appended claims, the singular forms“a,” “an” and “the” include plural referents unless the context clearlydictates otherwise. Ranges may be expressed herein as from “about” oneparticular value, and/or to “about” another particular value. When sucha range is expressed, another embodiment includes from the oneparticular value and/or to the other particular value. Similarly, whenvalues are expressed as approximations, by use of the antecedent“about,” it will be understood that the particular value forms anotherembodiment. It will be further understood that the endpoints of each ofthe ranges are significant both in relation to the other endpoint, andindependently of the other endpoint.

“Optional” or “optionally” means that the subsequently described eventor circumstance may or may not occur, and that the description includesinstances where said event or circumstance occurs and instances where itdoes not.

Throughout the description and claims of this specification, the word“comprise” and variations of the word, such as “comprising” and“comprises,” means “including but not limited to,” and is not intendedto exclude, for example, other additives, components, integers or steps.“Exemplary” means “an example of” and is not intended to convey anindication of a preferred or ideal embodiment. “Such as” is not used ina restrictive sense, but for explanatory purposes.

Disclosed are components that can be used to perform the disclosedmethods and systems. These and other components are disclosed herein,and it is understood that when combinations, subsets, interactions,groups, etc. of these components are disclosed that while specificreference of each various individual and collective combinations andpermutation of these may not be explicitly disclosed, each isspecifically contemplated and described herein, for all methods andsystems. This applies to all aspects of this application including, butnot limited to, steps in disclosed methods. Thus, if there are a varietyof additional steps that can be performed it is understood that each ofthese additional steps can be performed with any specific embodiment orcombination of embodiments of the disclosed methods.

The present methods and systems may be understood more readily byreference to the following detailed description of preferred embodimentsand the Examples included therein and to the Figures and their previousand following description.

Referring now to FIG. 1, example methods of performingnavigation-assisted medical procedures are described. It should beunderstood that the navigation-assisted medical procedures can be atleast partially performed by at least one processor (described below).Additionally, the navigation-assisted medical procedures can optionallybe implemented within a cloud computing environment, for example, inorder to decrease the time needed to perform the algorithms, which canfacilitate visualization of the prior analysis on real-time images.Cloud computing is well-known in the art. Cloud computing enablesnetwork access to a shared pool of configurable computing resources(e.g., networks, servers, storage, applications, and services) that canbe provisioned and released with minimal interaction. It promotes highavailability, on-demand self-services, broad network access, resourcepooling and rapid elasticity. It should be appreciated that the logicaloperations described herein with respect to the various figures may beimplemented (1) as a sequence of computer implemented acts or programmodules (i.e., software) running on a computing device, (2) asinterconnected machine logic circuits or circuit modules (i.e.,hardware) within the computing device and/or (3) a combination ofsoftware and hardware of the computing device. Thus, the logicaloperations discussed herein are not limited to any specific combinationof hardware and software. The implementation is a matter of choicedependent on the performance and other requirements of the computingdevice. Accordingly, the logical operations described herein arereferred to variously as operations, structural devices, acts, ormodules. These operations, structural devices, acts and modules may beimplemented in software, in firmware, in special purpose digital logic,and any combination thereof. It should also be appreciated that more orfewer operations may be performed than shown in the figures anddescribed herein. These operations may also be performed in a differentorder than those described herein.

Referring now to FIG. 1, a flow diagram illustrating example operations100 for performing navigation-assisted medical procedures is shown. Atstep 102 location information of an item of interest located within atleast a portion of a subject is obtained. For example, obtaininglocation information of the item of interest can comprise obtaining animage of at least a portion of the item of interest such as atwo-dimensional (2D) or a three-dimensional (3D) image. In one aspect,obtaining a three-dimensional image of at least a portion of the item ofinterest comprises obtaining the three-dimensional image using atargeted imaging technique. Such targeted techniques can include usingone or more of Positron Emission Tomography/Computed Tomography(PET/CT), Single Photon Emission Computed Tomography/Computed Tomography(SPECT/CT), Positron Emission Tomography Magnetic Resonance Imaging (PETMRI), fluorescence spectroscopy, and the like. In various embodiments,the targeted imaging techniques can comprise injecting the subject witha material and obtaining a three-dimensional image of at least a portionof the item of interest using one or more cameras that detect theinjected material. For example, the injected materials can comprise oneor more of Technetium-99, F18, zirconium 89, iodine 123, iodine 124,iodine 125, copper 64, gallium 67, gallium 68, lutetium 177, xenon 133,indium 111, and the like. In other aspects, obtaining athree-dimensional image of at least a portion of the item of interestcomprises obtaining the three-dimensional image of at least a portion ofthe item of interest using a non-targeted imaging technique. Forexample, the non-targeted imaging techniques can include using one ormore of X-Ray, ultrasound, CT, MRI, and the like. The item of interestcan be any tissue, organ, growth or abnormality, bone section, and thelike. For example, obtaining location information of an item of interestlocated within at least a portion of a subject can comprise obtaininglocation information of a tumor within at least a portion of a subject,cancerous tissue within at least a portion of a subject, an organ withina body of the subject, a bone section used in bone transplant surgery,and the like. In one aspect, obtaining location information of an itemof interest located within at least a portion of a subject comprisesobtaining location information of the item of interest within a body ofthe subject, location information of the item of interest within anorgan of the subject, location information of the item of interestwithin a tissue specimen, and the like. For example, when obtaininglocation information of the item of interest within an organ of thesubject, the organ may be contained within a body of the subject (e.g.,when performing a biopsy or surgery), or the organ may have been removedfrom a body of the subject (e.g., pathology).

At step 104 of an exemplary method of performing a navigation-assistedmedical procedure, position information of a moveable device is sensed.Such sensing may active or passive sensing. In various embodiments, themoveable device can comprise a biopsy needle, a scalpel, a pathologywand, a locator wand, and the like. Alternatively or additionally, themoveable device can be a bone section that is being aligned with or in abone transplant surgery. In one aspect, the position information of themoveable device is sensed using a three-dimensional sensor. For example,a Kinect™ (Microsoft Corporation) or Kinect™-type depth sensor can beused to obtain the position information of the moveable device. In otherinstances, one or more cameras (infrared or visible), accelerometers,gyroscopes, and the like can be used to obtain location information ofthe moveable device. In various aspects, the three-dimensional sensor islocated on at least one of the moveable device, eyeglasses of a personin view of the moveable device, or anywhere the three-dimensional sensorcan sense the moveable device. In one aspect, the three-dimensionalsensor for obtaining location information of the moveable device can belocated on augmented reality eyeglasses such as, for example, GoogleGlass™.

At step 106, a relative position of the moveable device to the item ofinterest can be determined using the location information of the item ofinterest and the position information of the moveable device.Determining the relative position of the moveable device to the item ofinterest using the location information of the item of interest and theposition information of the moveable device can be at least partiallyperformed using one or more computing devices, such as the one describedherein.

At step 108, feedback is provided based on the relative position of themoveable device to the item of interest that can be used to change therelative position of the moveable device to the item of interest. Invarious aspects, providing feedback based on the relative position ofthe moveable device to the item of interest that can be used to changethe relative position of the moveable device to the item of interest cancomprise acquiring a real-time image of the at least a portion of thesubject, wherein the acquired real-time image is referenced to the samecoordinate system as the location information of the item of interestand the position information of the moveable device; and displaying thereal-time image of the at least a portion of the subject furthercomprising the location information of the item of interest and theposition information of the moveable device super-imposed on thereal-time image of the at least a portion of the subject. For example,in one embodiment acquiring a real-time image of the at least a portionof the subject can be performed using at least a pair of stereoelectro-optical cameras. Referencing the real-time image to the samecoordinate system as the location information of the item of interestand the position information of the moveable device can comprise placingone or more fiducials on or around the at least a portion of thesubject. In one aspect, the fiducials can comprise visual markers thatcan be detected by the cameras. In other aspects, the fiducials can beradioactive markers or other non-visual markers that can not only bedetected by the cameras but also be detected by PET/CT, SPECT/CT, PETMRI, fluorescence spectroscopy, and the like so that locationinformation of the item of interest can be obtained either continuouslywhile the moveable device is being position in reference to the item ofinterest or on an interment basis and the location information of theitem of interest and the real-time image can be reference to the samecoordinate system. In one aspect, the fiducials comprise both, visualand non-visual markers. Generally, the real-time image of the at least aportion of the subject further comprising the location information ofthe item of interest and the position information of the moveable devicesuper-imposed on the real-time image of the at least a portion of thesubject will be displayed on a display that is visible to a person thatis moving the moveable device. For example, in one aspect, the positioninformation of the moveable device super-imposed on the real-time imageof the at least a portion of the subject is displayed to the personusing augmented reality glasses that are worn by the person. Providingfeedback can, in some embodiments, comprise providing audible or hapticfeedback that is used to move the moveable device closer to the item ofinterest. For example, the audible feedback may comprisecomputer-generated voice commands. In one aspect, the audible feedbackcomprises computer-generated noises that change as the moveable devicemoves closer to or further away from the item of interest. For example,the noise may get louder as the movable device gets closer to the itemof interest, or it may get quieter. Similarly, a series of clicks mayget more frequent and or louder as the movable device gets closer to theitem of interest, or vice versa. These are only examples of feedbackthat can be provided based upon the movement of the moveable devicerelative to the item of interest and it is to be appreciated that anyform of feedback is contemplated in embodiments of this invention. Inone embodiment, the feedback based on the relative position of themoveable device to the item of interest that can be used to change therelative position of the moveable device to the item of interestcomprises providing a control signal that is used to provide roboticcontrol of the moveable device; alternatively this can be used as avocal comment to the person holding the movable device. Using the methoddescribed above, the moveable device (e.g., biopsy needle, pathologywand, scalpel, bone section, etc.) can be inserted into the item ofinterest, wherein the feedback can provide an indication of a depth andposition that the moveable device has been inserted into the item ofinterest.

Referring now to FIG. 2, a flow diagram illustrating example operations200 for performing navigation-assisted medical procedures is shown. Thisexemplary process 200 includes the steps of flow chart 100, above, butfurther includes the step (step 202) of re-registration of the locationinformation of the item of interest located within at least a portion ofa subject. This step comprises obtaining location information of theitems of interest either during or after a procedure involving themoveable device. Re-registration can be performed using the same devicesand techniques as described in step 102, above, to initially determinethe location of the item of interest. Re-registration may occur aftermovement of the item of interest, the at least a portion of the subject,or the subject, after removal of at least a portion of the item ofinterest from the at least a portion of the subject, and the like. Forexample, if the item of interest is cancerous tissue and the movabledevice is a scalpel, the surgeon may want to re-register the item ofinterest after the initial surgical procedure to make sure that all ofthe cancerous tissue has been removed. In one aspect, re-registrationoccurs dynamically while the medical procedure is being performed.

FIG. 3 illustrates yet another embodiment of a flow diagram illustratingexample operations 300 for performing navigation-assisted medicalprocedures. This exemplary process 300 includes the steps of flow chart100, above, but further includes the step (step 302) of estimating thelocation information of the item of interest located within at least aportion of a subject based on modeling the at least a portion of asubject and using an algorithmic deformation model to handle theunobserved motion within the body of the subject. Such movement of theitem of interest may be cause by palpation, movement, repositioning ofthe subject or the at least a portion of a subject and the like. Forexample, as shown in FIGS. 4A and 4B, location information of the itemof interest 404 located within at least a portion of a subject 406 isinitially obtained (see FIG. 4A). In FIG. 4B, the same portion of thesubject 406 undergoes deformation or movement. Such deformation ormovement causes movement and possibly deformation of the item ofinterest 404 within the portion of the subject 406.

FIG. 5 illustrates an exemplary process diagram according to embodimentsof the present invention. As shown in the process diagram, a 3D tissueor lymphatic map 502 can be created; a dynamic 3D scene is alsodeveloped (step 504); a soft tissue motion engine (506) can be used toestimate how the item of interest moves or deforms within at least aportion of the subject; tool localization (508) comprises tracking thelocation of the wand, scalpel, biopsy needle, bone segment, and thelike; registration (510) involves bringing the 3D tissue or lymphaticmap, the 3D scene, estimation location changes for the item of interestand the tool localization together to provide to an augmented realityengine 512, which can visual, vocal, haptic, 3D and the like feedback tothe professional 514 performing the procedure. The professional can alsoprovide information about any tumors or other items of interest backdiscovered while performing the procedure to the 3D map so that theprocedure can be optimized thereby requiring fewer follow-on procedures.

3D tissue or lymphatic map creation at step 502 can involve one or moreof targeted or non-targeted preoperative imaging; marking of an item ofinterest and 3D map generation. Targeted or non-targeted imaging cancomprise CT, MRI, PET/CT, SPECT/CT, ultrasound, and the like. Marking ofthe item of interest can comprise marking the item of interest using,for example, radioactive, magnetic, RFID, and the like marker placementon, in or near the item of interest. 3D map generation can involvetriangulation of regions, radio frequency (RF) time of flight, and 3Dsurface modeling using CT, MRI, SPECT/CT, and the like.

Developing a dynamic 3D scene (step 504) can involve depth sensing viainfrared imaging using, for example, structured light concept or time offlight principals. Establishing the dynamic 3D scene can also involvestatic marker (fiducial) placement and tracking using, for example, astatic multi-camera setup and real-time detection, tracking andtriangulation.

The soft tissue motion engine (506) comprises tissue density models ofstatic 3D tissue, lymphatic map and bone. 3D surface sensing is used tosense movement of the surface of the at least portion of the subject andmathematical models are used to estimate the effect surface motion hason the item of interest. Therefore, an estimate of how the item ofinterest moves or deforms within at least a portion of the subject canbe developed.

Tool localization 508 comprises positioning and orientation estimationof a tool used by a professional such as a wand, scalpel, needle, probe,and the like or can even be used to estimate the location of a bonesegment. This can be accomplished in at least two ways

-   -   one way comprises tracking and triangulation of visual markers        on the tool or bone segment and the other way comprises special        tools designed with inertial sensors such as accelerometers,        gyroscopes, magnetometer, and the like.

Registration 510, as described above, comprises bringing the 3D tissueor lymphatic map, the 3D scene, estimation location changes for the itemof interest and the tool localization together to provide to anaugmented reality engine 512 so that the professional can be providedwith 3D localization of the item of interest, the scene, etc. Theprofessional can be provided with one or more visual vocal and hapticfeedback. Visual feedback can be in the form of, for example, aperspective 3D map projection to goggles or glasses worn by theprofessional, scene projection of a 3D tissue map via a projector, andthe like. Vocal feedback can provide voice or computer generated verbaldirections to the professional. Haptic feedback can be multipoint hapticfeedback to provide, for example, vibration to denote distance andposition of the tool to the item of interest, distance encoded vibrationfrequency, and the like.

Also, as noted above, the professional can provide feedback during orafter the procedure. For example, during cancer surgery the professionalcan mark new unobserved tumors via radioactive probes or palpation, 3Dmapping for the new tumor, or marking resected tissue for 3D mapremoval. For example, the professional can provide vocal or touchfeedback to the system.

When the logical operations described herein are implemented insoftware, the process may execute on any type of computing architectureor platform. For example, referring to FIG. 6, an example computingdevice upon which embodiments of the invention may be implemented isillustrated. In particular, at least one processing device describedabove may be a computing device, such as computing device 600 shown inFIG. 6. The computing device 600 may include a bus or othercommunication mechanism for communicating information among variouscomponents of the computing device 600. In its most basic configuration,computing device 600 typically includes at least one processing unit 606and system memory 604. Depending on the exact configuration and type ofcomputing device, system memory 604 may be volatile (such as randomaccess memory (RAM)), non-volatile (such as read-only memory (ROM),flash memory, etc.), or some combination of the two. This most basicconfiguration is illustrated in FIG. 6 by dashed line 602. Theprocessing unit 606 may be a standard programmable processor thatperforms arithmetic and logic operations necessary for operation of thecomputing device 600.

Computing device 600 may have additional features/functionality. Forexample, computing device 600 may include additional storage such asremovable storage 608 and non-removable storage 610 including, but notlimited to, magnetic or optical disks or tapes. Computing device 600 mayalso contain network connection(s) 616 that allow the device tocommunicate with other devices. Computing device 600 may also have inputdevice(s) 614 such as a keyboard, mouse, touch screen, etc. Outputdevice(s) 612 such as a display, speakers, printer, etc. may also beincluded. The additional devices may be connected to the bus in order tofacilitate communication of data among the components of the computingdevice 600. All these devices are well known in the art and need not bediscussed at length here.

The processing unit 606 may be configured to execute program codeencoded in tangible, computer-readable media. Computer-readable mediarefers to any media that is capable of providing data that causes thecomputing device 600 (i.e., a machine) to operate in a particularfashion. Various computer-readable media may be utilized to provideinstructions to the processing unit 606 for execution. Common forms ofcomputer-readable media include, for example, magnetic media, opticalmedia, physical media, memory chips or cartridges, a carrier wave, orany other medium from which a computer can read. Examplecomputer-readable media may include, but is not limited to, volatilemedia, non-volatile media and transmission media. Volatile andnon-volatile media may be implemented in any method or technology forstorage of information such as computer readable instructions, datastructures, program modules or other data and common forms are discussedin detail below. Transmission media may include coaxial cables, copperwires and/or fiber optic cables, as well as acoustic or light waves,such as those generated during radio-wave and infra-red datacommunication. Example tangible, computer-readable recording mediainclude, but are not limited to, an integrated circuit (e.g.,field-programmable gate array or application-specific IC), a hard disk,an optical disk, a magneto-optical disk, a floppy disk, a magnetic tape,a holographic storage medium, a solid-state device, RAM, ROM,electrically erasable program read-only memory (EEPROM), flash memory orother memory technology, CD-ROM, digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices.

In an example implementation, the processing unit 606 may executeprogram code stored in the system memory 604. For example, the bus maycarry data to the system memory 604, from which the processing unit 606receives and executes instructions. The data received by the systemmemory 604 may optionally be stored on the removable storage 608 or thenon-removable storage 610 before or after execution by the processingunit 606.

Computing device 600 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by device 600 and includes both volatile and non-volatilemedia, removable and non-removable media. Computer storage media includevolatile and non-volatile, and removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer readable instructions, data structures, program modules orother data. System memory 604, removable storage 608, and non-removablestorage 610 are all examples of computer storage media. Computer storagemedia include, but are not limited to, RAM, ROM, electrically erasableprogram read-only memory (EEPROM), flash memory or other memorytechnology, CD-ROM, digital versatile disks (DVD) or other opticalstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or any other medium which can be used tostore the desired information and which can be accessed by computingdevice 600. Any such computer storage media may be part of computingdevice 600.

It should be understood that the various techniques described herein maybe implemented in connection with hardware or software or, whereappropriate, with a combination thereof. Thus, the methods andapparatuses of the presently disclosed subject matter, or certainaspects or portions thereof, may take the form of program code (i.e.,instructions) embodied in tangible media, such as floppy diskettes,CD-ROMs, hard drives, or any other machine-readable storage mediumwherein, when the program code is loaded into and executed by a machine,such as a computing device, the machine becomes an apparatus forpracticing the presently disclosed subject matter. In the case ofprogram code execution on programmable computers, the computing devicegenerally includes a processor, a storage medium readable by theprocessor (including volatile and non-volatile memory and/or storageelements), at least one input device, and at least one output device.One or more programs may implement or utilize the processes described inconnection with the presently disclosed subject matter, e.g., throughthe use of an application programming interface (API), reusablecontrols, or the like. Such programs may be implemented in a high levelprocedural or object-oriented programming language to communicate with acomputer system. However, the program(s) can be implemented in assemblyor machine language, if desired. In any case, the language may be acompiled or interpreted language and it may be combined with hardwareimplementations.

FIG. 7 illustrates an exemplary overview system for performingnavigation-assisted medical procedures. In one embodiment, the systemcomprises a device 702 that obtains location information of an item ofinterest 704 located within at least a portion of a subject 706. In oneaspect, the device 702 that obtains location information of an item ofinterest 704 located within at least a portion of a subject 706comprises a device that obtains images using targeted imagingtechniques. For example, the device 702 can be one or more of a PET/CTscanner, a SPECT/CT camera, a PET MRI system, a florescence detectionsystem, and the like. In various aspects, the device 702 that obtainsimages using targeted imaging techniques includes one or more camerasthat can detect a material that has been injected into the item ofinterest or the subject. For example, the device 702 can be a camera,such as a gamma camera, that can detect one or more of Technetium-99,F18, zirconium 89, iodine 123, iodine 124, iodine 125, iodine 131,copper 64, gallium 67, gallium 68, lutetium 177, xenon 133, indium 111,and the like. In other aspects, the device 702 that obtains locationinformation of an item of interest located within at least a portion ofa subject can comprise a device that obtains images using non-targetedimaging techniques. For example, the device 702 can be one or more of anX-Ray system, an ultrasound scanner, a CT, an MRI, and the like. Theitem of interest 704 can be one or more of a tumor within at least aportion of a subject 706, cancerous tissue within at least a portion ofa subject 706, a tissue sample of the subject, an organ within a body ofthe subject, a bone segment for a bone transplant, and the like. Forexample, the item of interest 704 can be located within a tissuespecimen, within a body of the subject, an organ of the subject, and thelike. If the item of interest 704 is an organ, the organ may becontained within a body of the subject or the organ may have beenremoved from a body of the subject.

Further comprising the system of FIG. 7 is a sensor 708 that sensesposition information of a moveable device 710. The sensor 708 can be anactive or a passive sensor. For example, the sensor 708 can be athree-dimensional sensor. For example, a Kinect™(Microsoft Corporation)or Kinect™-type depth sensor can be used to obtain the positioninformation of the moveable device 710. In other instances, the sensor708 can be one or more cameras (infrared or visible), accelerometers,gyroscopes, and the like that can be used to obtain location informationof the moveable device 710. In various embodiments, sensor 708 can belocated on at least one of the moveable device 710, eyeglasses of aperson in view of the moveable device, or anywhere the sensor 708 cansense the moveable device 710. The moveable device 710 can be, forexample, a biopsy needle, a scalpel, a pathology wand, a locator wand, abone segment, and the like.

Further comprising the exemplary system of FIG. 7 is a computing device600, as described above. The computing device 600 receives the locationinformation of the item of interest 704 from the device 702 and theposition information of the moveable device 710 from the sensor 708 andexecutes computer-readable instructions that determine a relativeposition of the moveable device 710 to the item of interest 704 usingthe location information of the item of interest 704 from the device 702and the position information of the moveable device 710 from the sensor708, and provides feedback based on the relative position of themoveable device 710 to the item of interest 704 that can be used tochange the relative position of the moveable device 710 to the item ofinterest 704. The computing device 600 can communicate with the sensor708 and the device 702 using a network 712. The network 712 can comprisehardware and software. The computing device 600 can connect with thesensor 708 and the device 702 using wires (including fiber optic),wirelessly, or combinations of wired and wirelessly. In one aspect, thecomputing device 600 executes computer-readable instructions that causethe computing device 600 to provide audible feedback that is used tomove the moveable device 710 closer to the item of interest 704. Forexample, the audible feedback may comprise computer-generated voicecommands, computer-generated noises or haptic feedback that change asthe moveable device 710 moves closer to or further away from the item ofinterest 704. Though not shown in FIG. 7, in one embodiment the systemcan further comprise a robotic surgical device, wherein the computingdevice 600 executes computer-readable instructions that cause thecomputing device 600 to provide a control signal that is used to providerobotic control of the moveable device 710.

FIG. 8 illustrates an embodiment of a system for performingnavigation-assisted medical procedures further comprising one or morecameras 802 in communication with the computing device 600. The one ormore cameras 802 acquire a real-time image of the at least a portion ofthe subject 706, wherein the acquired real-time image is referenced tothe same coordinate system as the location information of the item ofinterest 704 and the position information of the moveable device 710. Inone embodiment, the system of FIG. 8 can further comprise a display 804in communication with the computing device 600, wherein the real-timeimage of the at least a portion of the subject 706 is provided to thecomputing device 600 and the location information of the item ofinterest 704 and the position information of the moveable device 710 issuper-imposed on the real-time image of the at least a portion of thesubject 706 and displayed on the display 804. As shown in FIG. 8, in oneaspect the display 804 can comprise augmented reality glasses.Embodiments of the system may further comprise one or more fiducials806, wherein referencing the acquired real-time image to the samecoordinate system as the location information of the item of interest704 and the position information of the moveable device 710 is performedusing the one or more fiducials 806 on or around the at least a portionof the subject 706. In one aspect, at least one of the one or morefiducials 806 can comprise a radioactive fiducial. In various aspects,the computing device 600 can provide feedback that indicates a depththat the moveable device 710 has been inserted into the item of interest704. This can be done using audible feedback, visual feedback or acombination or audible and visual feedback.

The systems of FIGS. 7 and 8 enable re-registration of the item ofinterest 704. Re-registration comprises obtaining location informationof the items of interest 704 either during or after a procedureinvolving the moveable device 710. Re-registration can be performedusing the same devices and techniques as described in FIGS. 7 and 8 toinitially determine the location of the item of interest 704.Re-registration may occur after movement of the item of interest 704,the at least a portion of the subject 706, or the subject, after removalof at least a portion of the item of interest 704 from the at least aportion of the subject 706, and the like. For example, if the item ofinterest 704 is cancerous tissue and the movable device is a scalpel,the surgeon may want to re-register the item of interest 704 after theinitial surgical procedure to make sure that all of the cancerous tissuehas been removed. In one aspect, re-registration occurs dynamicallywhile the medical procedure is being performed.

EXAMPLES

A. Pathological Specimen Sampling

Pathologic staging of solid tumors involves determining the presence andextent of disease. Precise specimen processing is critical forestablishing patient customized management plans that may indicate theneed for post-operative chemotherapy and/or radiation therapy. Failureto identify malignant involvement in tissues can lead to themisdiagnosis and mismanagement of patients, leading to undesirableoutcomes. Furthermore, correctly determining the extent of diseaseinvolves accurate assessment of the tumor's size, and the presence orabsence of metastatic disease involving lymph nodes and distant tissues.Hence, the pathologist's assessment is crucial, and it plays a key rolein basing future treatment options for patients.

FIGS. 9A-9E illustrate an exemplary integrated process for targetedspecimen 902 analysis. As shown in FIG. 9A, the tissue in this scenariois injected with a marker such as, for example, ⁹⁹mTc for nuclearimaging. In FIG. 9B, a device 904 such as, for example a portable gammacamera and planar lymphoscintigraphy are incorporated to permit 3D tumorlocalization in the specimen 902. After surgical resection (FIG. 9C),additional images 906 are taken of the specimen 902 (FIG. 9E). Utilizingaugmented reality, 3D enhanced pathologic specimen sampling is performedfor targeting the tumor as shown in FIG. 9E.

Embodiments of the system described herein provide an integrated systemto improve the quality of analysis of the resected tumor specimen thatprovides the pathologist with the latest technological and scientificadvancements. The developed system improves pathological detectionefficiency and accuracy. It minimizes the occurrence of leaving tumorsunder-sampled, and thus reduces the incidence of misdiagnosis. Thesystem is flexible in that it can be used for intra-operative marginassessment and postoperative specimen processing. Embodiments of thesystem provide image-guided cancer pathology by integrating amulti-camera tracking and navigation system with a portable Gamma Cameraand SPECT/CT for low energy radio-isotopes and PET/CT imaging for highenergy radio-isotopes. Images obtained can be used to generate anaugmented visualization in a wearable product, such as Google Glass™,for guiding the pathologist by visualizing the location and extent ofthe cancerous tissue.

Currently, pathologists and pathologist's assistants (PA) rely on visualand tactile cues to identify and locate tumors and lymph nodes inresected tissue specimens. The process, or grossing, of a tissuespecimen can be time consuming and is prone to sampling error. It hasfrequently been associated with “finding a needle in a haystack.”Inaccurate processing of a tissue specimen may result in the incorrectpathologic staging being given. This can be the consequence ofincomplete sampling of a lymph node, completely missing the tumor, or afailure to recognize the true relationship of the tumor to the closetmargin of resection by sectioning the specimen in the wrong plane. Thus,it is highly desirable that the pathologist “can see” the cancer in itsentirety in order to optimize treatment and offer the patient the bestpossible outcome.

Embodiments of the system described herein guide the pathologist/PA towhere the tissue needs sampling. In cases of cancer of the head andneck, testes, prostate, melanoma, bladder, rectum, pancreas, stomach,and colon, accurate assessment of lymph nodes for metastatic disease isa must. Bringing the intra/postoperative images of the patient and theresected tissue to the pathologist's/PA's eye saves time while markedlyincreasing accuracy. The end result is improved treatment decisionmaking.

A standard 2D image of a large specimen doesn't accurately reflect thenumber of tumors, their orientation, or their relationship to thenearest surgical margin of resection. Embodiments of the systemdescribed herein provide the pathologist/PA with a 3D view of thetumor(s) being targeted. This enables him/her with the ability toaccurately identify and precisely sample the tissue, including theevaluation of the surgical margin of resection. This is especiallyuseful in wide local excision of melanoma. Additionally, due to thetechnical advantages of this system, tissue sampling becomes moreefficient and may provide intra-operative feedback to the surgeon in amanner that is more accurate than current intra-operative tissueanalysis.

Described below is a non-limiting exemplary test of an embodiment of asystem described herein. In FIG. 10A, a phantom bench top study isconducted with the fiducials 1002 placed on the phantom table 1004 andthe gamma camera head 1006. FIG. 10B illustrates an exemplar gamma scansimulating breast sentinel node and head/neck sentinel nodes. FIG. 10Cillustrates back-projected spheres (16 mL, 4 mL, and 1 mL) superimposedon one of the collected images, as seen through a moving camera. Thethree spheres, simulating separately sized tumors (1 mL, 4 mL and 16mL), were injected with Technetium 99m and placed inside a glass phantom1008. The glass phantom 1008 represents the entire resected specimencontaining the foci, or spheres, of malignant disease. Fiducials 1002were set around the phantom 1008 and on the gamma camera 1006 to recoverthe 3D geometry. Eight images were collected with the gamma camera fromdifferent viewing angles at 1.07″ distance from the phantom. Resultsshow successful recovery of the 3D sphere locations, as well as preciseback-projections to the image acquired from a moving camera.

FIG. 11 illustrates one type of setup that can be used to performaspects of embodiments of the invention. In this setup, the proposedsystem is comprised of stereo electro-optical (EO) camera pair 1102 withoverlapping field-of-views, medical imaging device 1104, a pathologylocator wand 1106 and augmented reality glasses (e.g., Google Glass™)1108, which can be worn by the pathologist. The EO cameras 1102 are usedto detect and track a set of fiducials 1110 placed on the specimen tray1112. The gamma camera, or other imaging modality 1104, provides thelocations of lymphatic and cancerous tissues within the resectedspecimen 1114. In this setting, the augmented reality glasses 1108 canbe used to generate an augmented reality by superimposing the locationof the targeted tissues and the tip of the pathology locator wand 1106onto the specimen 1114, as seen by the pathologist wearing the augmentedreality glasses 1108. The wand 1106 can be used to pinpoint the targetedcancerous tissue inside the resected specimen 1114 and enable marginassessment quickly. A second proposed setup can generate the augmentedreality by projecting the cancerous tumors onto the augmented realityglasses 1108 display without the use of the stereo camera pair 1102.This setting can provide visualization without the wand 1106. Regardlessof the method used, upon removal of the targeted tumor(s) from thespecimen 1114, the specimen 1114 can be reimaged to ensure preciseremoval of the targeted tumor(s) from the specimen 1114. In one aspect,the pathologist wearing the optics is able to point to the targetedtumor with the wand 1106 to within 1 mm of accuracy; thereby, ensuringaccurate specimen 1114 processing. In one embodiment, distanceindicators on the wand 1106 can be used to visually guide thepathologist.

In both of the above-described exemplary setups, each camera 1102resides in their own reference frame and requires a referencing schemeto bring each of them into the same coordinate system. For pathologystudies, two potential static reference frames that can be used includethe stereo camera pair 1102 and the specimen tray 1112. In order tofacilitate the referencing of all system components to a staticreference frame, a shared set of radioactive fiducials 1110 can beplaced on the tray 1112 that can be captured by the gamma camera image,augmented reality glasses 1108 and the stereo camera pair 1102 (ifused). The initialization of the process starts with estimating theinterior and exterior orientation parameters of the cameras prior tosurgery, which will later be used for registration, detection andtracking purposes.

With the proper setup established, tissue analysis can be performed.Prior to the patient's operation, the patient can be injectedintravenously with a nuclear isotope, such as Technetium 99. The isotopeattaches to an antibody and the antibody, peptide and alike goes to thelymph nodes. While the tumor may not be directly detected or targeted assuch, the potential location of the tumor and the isotope and itscarrier can be injected around it for lymphatic mapping. The actualradio-isotope administered will be determined by the tumor type beingtargeted. During the surgery, the specimen(s) can be surgically resectedfollowing standard oncologic techniques. Once it has been removed fromthe patient, the tissue(s) can be placed on the specimen tray 1112containing four or more embedded and shared fiducials 1110 at fixedpositions that will be within the nuclear imaging field of view asmentioned above. The option for imaging modality includes, but is notlimited to, the Gamma camera, SPECT/CT, PET/CT and micro SPECT/CT. Theparticular imaging method used for a given case can be guided by theparticular radio-isotope administered. The 3D locations on the fiducials1110 can be measured and registered to the augmented reality glasses1108 camera and the EO stereo system 1102 (if used) by photogrammetrictriangulation. The resected specimen containing the tumor(s), andaccumulated isotope within can be visualized on the acquired images froma minimum of two different orientations with a baseline betweenacquisition positions. Using a pre-established model that adoptsorthographic and para-perspective projections, the 3D positions of thetargeted tissues can be estimated through the same process used incomputing the 3D positions of shared fiducials.

A statistical software package can be used to analyze all the bench topand vivo data. The bench top measurements are expressed as mean±standarddeviation. The in vivo data can be analyzed using one-way analysis ofvariance (ANOVA) method, with the differences among the means evaluatedby the Tukey-Kramer multiple comparison test. A P-value of less than0.05 is considered statistically significant.

B. Image-Guided Cancer Surgery

Historically, surgery has offered the best opportunity for cure inpatients with solid malignancies. Previous optimal surgical approach tocancer resection was limited to visual and tactile cues in identifyingthe tumor's location and extent. Operations were based on surgicalanatomy and traditional planes of resection that cancer cells frequentlydisregarded. Thus, it became evident that the a challenge to cancersurgery is for the surgical team “see” all of the tumor(s), includingmetastases, to achieve the best resection and offer the patient the bestpossible outcome. Since the early 1990s, this concept has materializedby the incorporation of preoperative imaging, such as PET/CT andSPECT/CT. However, these imaging techniques do not provide a real-timeintra-operative assessment of tumor location or to indicate whether themalignancy has been completely resected.

Described herein are embodiments of an integrated system for real-timeintra-operative tumor tracking and surgical guidance. Embodiments ofthis invention deliver better science to the surgeon before, during andafter the surgery to help locate primary and metastatic tumors, guidecomplete excision of malignant tissues in a single surgery, and assesscompleteness of resection. As a result, cost-effective and personalizedpatient care, with improved outcomes, will be achieved while reducingthe chance of leaving residual tissue behind. One embodiment of thesystem integrates multi-camera tracking and navigation system withintra-operative portable gamma camera scans for generating augmentedvisualization for surgical guidance until no residual cancer remains.

FIGS. 12A-12H illustrate an integrated methodology and schema forimage-guided cancer surgery. In FIG. 12A, the tissue is injected with,for example, ^(99m)Tc for intra-operative imaging. In FIGS. 12B and 12C,preoperative gamma camera is incorporated for 3D lymphatic mapping,which in FIG. 12D, is projected onto augmented reality glasses (e.g.,Google Glass™) for surgical guidance. As shown in FIGS. 12E-12H,additional gamma camera images are scanned and residual cancerous tissueis resected until the cancer has been completely removed.

Accurate assessment of the extent of disease is the cornerstone tominimizing cancer recurrence rates and ultimately translates intoimproved overall outcomes. While significant advances have occurred inthe last two decades, cancer imaging is still primarily focused onpreoperative image acquisition. Currently, the only intra-operativesurgical tool that provides real-time cancer-specific information is thehand-held gamma probe. However, it does not provide informationregarding tumor margins. Furthermore, localizing small tumors with theprobe may take a significant amount of time due to the difficulty indirecting the probe with respect to the preoperative images. Therefore,embodiments of the system described herein provide real-time surgicalguidance system for tumor delineation, lymphatic mapping, and fornavigating the surgeon to missed residual tumor foci for resectionduring the initial operation. This can lead to fewer reoperations andimproved surgical cure rates, which can be up to 20% in breast cancer.

FIG. 13 illustrates an embodiment of a system for operating room (OR)setup for intra-operative imaging and 3D recovery and tracking. Thisembodiment is comprised of stereo electro-optical (EO) cameras 1302 withoverlapping field-of-views, one gamma camera 1306 and a display oraugmented reality glasses worn by the surgeon (not shown FIG. 13). TheEO cameras 1302 are used to detect and track a set of fiducials 1304 onthe patient and surgeon; the gamma camera 1306 provides the locations ofthe targeted tissue, and the display or augmented reality glasses areused to generate augmented reality by projecting these regions onto thesurgical scene as seen by the surgeon. Each one of these cameras residesin their own reference frame and requires an explicit referencing schemethat will bring them into the same frame. One stationary component ofthe setup that can used to establish the reference frame is the stereocamera 1302; therefore, all imaging devices can be referenced to thestereo system 1302. In order to facilitate this referencing, a sharedset of radioactive fiducials 1304 can be placed on the patient only atthe time of gamma scanning, and they will be captured in both the gammacamera image and the stereo system images. The initialization of thesurgical process starts with estimating the interior and exteriororientation parameters of the stereo cameras prior to surgery and willlater be used for registration, detection and tracking purposes.

Prior to preoperative gamma camera imaging, a plurality (e.g., four)shared fiducials 1304 are placed on or near the patient at positionsthat are within the gamma camera's 1306 field-of-view. The 3D locationson the fiducials 1304 are registered to the stereo system 1302 byphotogrammetric triangulation that estimates (X, Y, Z) of the fiducials1304 by a bundle adjustment process. The gamma scans of the tissueexposed to the Technetium 99m can then be acquired from a minimum of twodifferent orientations with some baseline between acquisition positions.Using a pre-established gamma camera model that adopts orthographicprojection, the 3D centers of the targeted tissues can be estimatedthrough the same process used in computing the 3D positions of theshared fiducials 1006. In order to establish a model for delineating themargin of resection for suspected cancers, the gamma camera 1306 imagetypically provides different margins at different contrast levels. Inorder for the surgeon to plan a safe margin of resection around thetumor, a maximum contrast level is used and the entire set of pixelswithin the margin from all acquired images is back-projected, which willgenerate a 3D volume. This can then be modeled by radon transform orconvex hull that can be used for generating augmented reality. Forcomputational efficiency, volumetric methods with voxel size adjustedaccording to the gamma camera pixel size are adopted. As noted abovewith reference to FIGS. 10A-10C, a successful benchtop study has beenconducted to prove the localization and back projection conceptintroduced above.

At times, multiple tumors will be present and one, or more, may beobstructed or hidden behind a more superficial tumor, which may lead toresidual cancer left behind if not identified. One approach is to use asurgical wand, such as a suction tip, to guide the surgeon to suchcancerous tissues prior to completing the surgery. This guidance allowsthe surgeon to perform a secondary check with the hand-held gamma probeto locate and make sure that the tissue is truly involved. In order toachieve this, the stereo camera system detects and tracks a set offiducials mounted onto the wand in real-time to estimate its tipposition and orientation. One possible arrangement is to place one ofthe fiducials on the wand base and the other towards the tip of the wandto define a 3D line for tracking purposes. Another arrangement is toorganize four fiducials on a planar patch mounted onto the base of thewand.

Prior to the operation, the relative positions of the fiducials can beregistered to the tip of the wand by the stereo system for accurateposition and orientation estimation. The system detects and tracks thefiducials by performing kernel-tracking algorithms modified byintroducing geometric constraints to improve geometric localization. Inaddition, spatiotemporal constraints are used to reduce thecomputational time and increase precision. The spatiotemporalconstraints are incorporated into the detection and tracking frameworkin the form of Markov Random Fields applied to a stack of consecutiveframes generating Markov Random Volumes. Upon tracking the positions,the orientation and the wand tip position can be estimated byintersecting tracked locations from stereo-cameras by bundle adjustment.Once the tip is located, the surgeon will have visual and auditoryguidance that will include verbal directions to move the wand tiptowards the target tissue. The auditory signals are referenced to thewand tip and are in the form of directional information, such as “left”and “right”, and distance information.

Visual feedback can be generated by back-projecting the 3D locations ofthe target tissues and the wand tip to an augmented reality displayinterface (e.g., Google Glass™). The back-projection requiresregistering the 3D position and orientation of the augmented realitydisplay interface to the stereo camera reference frame. Thisregistration requires real-time tracking of the augmented realitydisplay interface in stereo image pair. For achieving high precision inaugmented reality display interface tracking, a set of fiducials can bemounted on a surgical head-band worn by the surgeon, and recoverrelative geometry between them by triangulating matching fiducials. Thisprocess can be performed before the operation when the surgeon puts thehead-band and augmented reality display interface on. In order to assistaugmented reality display interface tracking, additional sensory datathat the augmented reality display interface (e.g., Google Glass™)provides, such as magnetometer and inertial measurements can be used.These measurements will serve as initial orientation information foriterative orientation estimation. Upon recovering the augmented realitydisplay interface (e.g., Google Glass™) orientation, the 3D positions ofwand tip and the target tissues, i.e. parathyroid adenomas, can beback-projected to the augmented reality display interface (e.g., GoogleGlass™) camera display unit to create an augmented reality that adjuststo the surgeons viewing angle to correctly resect the target tissue. Thegamma scanning, 3D localization and tracking followed by tumor resectioncan be iterated as many times as necessary until no residual tissue isleft.

FIG. 14 is an exemplary flow diagram that illustrates the integratednature and use of embodiments of the present invention in a medicalapplication. For example, a 3D model of an item of interest can becreated in nuclear medicine/imaging 1402 using devices and techniques asdescribed herein. The model created in nuclear medicine/imaging 1402 canbe used in surgery 1404 or pathology 1406 by a professional to bettersee the item of interest and to determine it margins or alignment.Overall, the embodiments of systems and methods described herein providea better tool for the medical professional and a better medicalexperience for the patient.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

1. A method of performing navigation-assisted medical procedurescomprising: estimating location information of an item of interestlocated within at least a portion of a subject using an algorithmicdeformation model based on acquired image information to estimateunobserved motion of the item of interest within at least the portion ofthe subject; sensing position information of a moveable device;determining a relative position of the moveable device to the item ofinterest using the estimated location information of the item ofinterest and the position information of the moveable device; andproviding directional and distance active feedback based on the relativeposition of the moveable device to the item of interest that can be usedto change the relative position of the moveable device to the item ofinterest during a navigation-assisted medical procedure, wherein theactive feedback changes as the moveable device moves closer to orfurther away from the item of interest, and wherein the distance activefeedback comprises distance encoded vibration frequencies providedthrough the moveable device.
 2. The method of claim 1, whereinestimating location information of the item of interest comprisesobtaining an image of at least a portion of the item of interest.
 3. Themethod of claim 1, wherein estimating location information of an item ofinterest located within at least a portion of a subject comprisesestimating location information of a tumor within at least a portion ofa subject, cancerous tissue within at least a portion of a subject, oran organ within a body of the subject.
 4. The method of claim 1, whereinestimating location information of an item of interest located within atleast a portion of a subject comprises estimating location informationof the item of interest within a body of the subject estimating locationinformation of the item of interest within an organ of the subject, orestimating location information of the item of interest within a tissuespecimen.
 5. The method of claim 1, further comprising inserting themoveable device into the item of interest, wherein the feedbackindicates a depth that the moveable device has been inserted into theitem of interest.
 6. The method of claim 1, wherein providing feedbackbased on the relative position of the moveable device to the item ofinterest that can be used to change the relative position of themoveable device to the item of interest further comprises: acquiring areal-time image of the at least a portion of the subject, wherein theacquired real-time image is referenced to the same coordinate system asthe location information of the item of interest and the positioninformation of the moveable device; and displaying the real-time imageof the at least a portion of the subject further comprising theestimated location information of the item of interest and the positioninformation of the moveable device super-imposed on the real-time imageof the at least a portion of the subject.
 7. The method of claim 1,wherein providing feedback based on the relative position of themoveable device to the item of interest that can be used to change therelative position of the moveable device to the item of interestcomprises providing one or more of visual, audible or haptic feedbackthat is used to move the moveable device closer to the item of interest.8. The method of claim 1, wherein providing feedback based on therelative position of the moveable device to the item of interest thatcan be used to change the relative position of the moveable device tothe item of interest comprises providing a control signal that is usedto provide robotic control of the moveable device.
 9. The method ofclaim 1, wherein sensing position information of the moveable devicecomprises actively or passively sensing position information of a biopsyneedle, a scalpel, a pathology wand, a locator wand, or a bone segment.10. The method of claim 1, wherein sensing position information of themoveable device comprises obtaining position information of the moveabledevice using a three-dimensional sensor.
 11. The method of claim 1,wherein determining the relative position of the moveable device to theitem of interest using the estimated location information of the item ofinterest and the position information of the moveable device isperformed using a computing device.
 12. The method of claim 1, furthercomprising re-registration of the estimated location information of theitem of interest located within at least a portion of a subject.
 13. Themethod of claim 1, further comprising estimating movement or deformationof the item of interest caused by movement or palpation of the at leasta portion of the subject, wherein the estimated location information ofthe item of interest is modified by the estimated movement ordeformation of the item of interest.
 14. (canceled)
 15. (canceled) 16.(canceled)
 17. (canceled)
 18. (canceled)
 19. (canceled)
 20. (canceled)21. (canceled)
 22. (canceled)
 23. (canceled)
 24. (canceled) 25.(canceled)
 26. (canceled)
 27. (canceled)
 28. (canceled)